Simulated infrared material combination using neural network

ABSTRACT

Mipping systems and methods are disclosed. For example, a mipping system can include processing circuitry configured to receive combinations of a plurality of pixels N at a time, each pixel having material codes directed to respective materials of the pixels, where the material codes relate to infrared properties of the respective materials, and N is a positive integer greater than 1; and train an artificial neural network having a classification space by providing respective neurons for each unique combination of material codes, and condition the artificial neural network so that the respective neurons activate when presented with their unique of material code combinations in order to create a combined set of material code parameters for accurate rendering of the mipped pixels.

INCORPORATION BY REFERENCE

This application claims the benefit of U.S. Provisional Application No.61/873,543 entitled “SIMULATED INFRARED MATERIAL COMBINATION USINGNEURAL NETWORK” filed on Sep. 4, 2013, the content of which isincorporated herein by reference in its entirety.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent the work is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

In a process called “mipping” or sub-sampling, a lower resolutionversion of a visual scene is created. This may be performed in order touse less memory when viewing a complicated scene from a distance. Theremay be multiple levels of mipping. For example, in one form, fouradjacent pixel colors are joined together to make one pixel having anaverage color of the four original pixels while in another form ninepixels are joined together to make one pixel having an average color ofthe nine original pixels.

In the use of Infrared (IR) materials, however, pixels are processeddifferently according to a concept known as “Material Coded Imagery”(MCI) which uses codes to represent different types of materials havingdifferent IR properties. Mipping in the infrared does notcombine/average different pixel colors as with visual mipping, butderives a single “material code” out of a set of known material codeshaving suitable properties that an infrared heating and image generation(sensor simulation) system uses to produce an image that looks correctfrom a distance and that can change into the individual elements as theview becomes closer.

SUMMARY

Various aspects and embodiments of the invention are described infurther detail below.

In an embodiment, a mipping system includes processing circuitryconfigured to: receive combinations of a plurality of pixels N at atime, each pixel having material codes directed to respective materialsof the pixels, where the material codes relate to infrared properties ofthe respective materials, and N is a positive integer greater than 1;and train an artificial neural network having a classification space byproviding a set of neurons that will recognize each unique combinationof material codes, and condition the artificial neural network so thatthe respective neurons activate when presented with their uniquematerial code combinations.

In another embodiment, a mipping method includes receiving combinationsof a plurality of pixels N at a time, each pixel having material codesdirected to respective materials of the pixels, where the material codesrelate to infrared properties of the respective materials, and N is apositive integer greater than 1; and training an artificial neuralnetwork having a classification space by providing respective neuronsfor each unique combination of material codes, and conditioning theartificial neural network so that the respective neurons activate whenpresented with their unique of material code combinations.

In yet another embodiment, an imaging device includes circuitryconfigured to process a plurality of input pixels taken N at a timeusing a pre-trained artificial neural network to create an output pixelfor each N input pixels, wherein the input pixels and the output pixelall include infrared information and more than three dimensions, eachpixel having a respective material code chosen from material codes of aclassification space; wherein N is a positive integer more than 1.

In still another embodiment, an imaging method includes receiving aplurality of pixels that each include infrared information and more thanthree dimensions, each pixel having a respective material code chosenfrom material codes of a classification space; and processing the pixelsusing a pre-trained artificial neural network to create an output pixelhaving a material code of the classification space and infraredinformation.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of this disclosure that are proposed as exampleswill be described in detail with reference to the following figures,wherein like numerals reference like elements, and wherein:

FIG. 1 is a block diagram of an example of a neural processing systemconfigured to train and use neural networks to ‘mip’ infrared visualdata.

FIGS. 2A and 2B depict examples of “mipping.”

FIG. 3 is a flowchart outlining an example set of operations fortraining and using an artificial neural network to mip infrared imagedata.

FIG. 4 is a flowchart outlining an example set of operations fornormalizing material coded information (MCI) codes.

FIG. 5 is a flowchart outlining an example set of operations forperforming a first training operation that creates hyper-points within aclassification space.

FIG. 6 is a pictorial representation of a first training operation thatcreates hyper-points within a classification space.

FIG. 7 is a pictorial example of a classification point withintwo-dimensions of a classification space.

FIG. 8 is a flowchart outlining an example set of operations forperforming a second training operation that creates hyper-rectangleswithin a classification space.

FIG. 9 is a pictorial representation of a second training operation thatcreates hyper-rectangles within a classification space.

FIG. 10 is a pictorial example of a classification area embedded withintwo-dimensions of a classification space.

DETAILED DESCRIPTION OF EMBODIMENTS

The disclosed methods and systems below may be described generally, aswell as in terms of specific examples and/or specific embodiments. Forinstances where references are made to detailed examples and/orembodiments, it is noted that any of the underlying principles describedare not to be limited to a single embodiment, but may be expanded foruse with any of the other methods and systems described herein as willbe understood by one of ordinary skill in the art unless otherwisestated specifically.

A pixel is the fundamental unit of an image used in computer graphics. Atexel (also known as a texture element, or texture pixel) is thefundamental unit of texture space. Textures are represented by arrays oftexels, just as pictures are represented by arrays of pixels. “Mipping”refers to the combination of multiple pixels or texels into a‘sub-sampled’ pixel/texel. Thus, for instance, four pixels/texels can becombined into one to create a view from a further distance.

For the purpose of this disclosure, the term “pixel” is to includepixels, texels or any other entity capable of representing an image.

Further, the ‘property boundaries’ (or ‘material codes’) of a singlematerial is a point (or “hyper-point” or “hyper-dimensional point” ifthe material is described in more than three dimensions). Each materialhas only one value albeit that value is described in multipledimensions. Generally, there is no range of settings per material. Whenmaterials are combined and presented to a neural network, eachcombination is also a hyper-point. However, it is to be recognized that,upon the application of fuzzy logic or some other fuzzy recognitionprocess, a hyper-point may be subsumed within a ‘hyper-range’ or‘hyper-rectangle” (more than three dimensions).

FIGS. 2A and 2B depict examples of mipping different materials. FIG. 2Ais provided to indicate that a collection of pixels 204 of differentmaterials each having different Material Coded Imagery (MCI)identifier/code can be combined into an output pixel 201 having an MCIidentifier/code that is an amalgam of the collection of pixels 204. FIG.2B is provided to indicate that collection of pixels can range greatly.That is, the output pixel 211 of FIG. 2B can be constructed, forexample, from a set of sixteen contiguous pixels 216 as well as a set offour contiguous pixels 214. FIG. 2B also shows that there may bemultiple levels of mipping, to portray imagery at different distancesfrom the observation point, for instance. The four mipped pixels at 214maybe further mipped to the single mipped, or multi-mipped, pixel at211. While the examples below will be limited to four pixels used tocreate an output pixel, in practice any number N of pixels (N being apositive integer greater than 1) can be mipped to create alower-resolution output pixel.

FIG. 1 is a block diagram of an example of a neural processing system100 configured to train and use neural networks to ‘mip’ infrared visualdata. As shown in FIG. 1, the neural processing system 100 includes aprocessor 110, a memory 120, normalization circuitry 130, phase 1training circuitry 140, phase 2 training circuitry 150 and mippingcircuitry 160. Within the memory 120 includes Infrared (IR) image data122, Material Code Information (MCI) data 124 for a variety of differentmaterials (e.g., painted objects, water and concrete), and a neuralmodel 126. The above components 110-160 are coupled together withcontrol/data bus 102. Although the example of FIG. 1 uses a bussedarchitecture controlled by a sequential instruction device, it is to beappreciated that there is a wide range of other architectures capable ofperforming equivalent tasks, such as a collection of dedicatedgates/circuits hardwired to one another using dedicated data and controllines, a collection of different computer-based processing systems, andso on.

FIG. 3 is a flowchart outlining the general operation of the processingsystem of FIG. 1 and includes the operations of normalization of MCIdata (S302), performing phase 1 training (S304), performing phase 2training (S306) and performing mipping operations of infrared imagesusing the resulting neural network (S308).

Returning to FIG. 1, under control of the processor 110, thenormalization circuitry 130 performs preprocessing andstandardization/normalization processes on the MCI data 124.

Preprocessing and Standardization/Normalization:

As discussed above, a set of infrared materials can be used to createnew materials from their combination, or to select one from thecombination of the existing materials that best represents thecombination. This is accomplished by training the neural network 126using all combinations of materials in order to allow the neural network126 to return the closest match in materials putting into consideringmany dimensions of physical properties that represent the materials.

In order to make the input information be in the correct form for inputto the neural network 126, preprocessing can be used tostandardize/normalize all the features of all the individual materialproperties and combined material properties such that the maximum andminimum values of all the dimensions are adjusted so that: Maximumvalue=1.0 and Minimum value=0.0.

In the present embodiment, for the materials processed, 118 dimensionswere chosen, which include:

Material Density

Material Emissivity

Specific Heat

Thermal Conductivity

Season of the year

Geometry information 1

Geometry information 2

Geometry information 3

Reflectivity: (110 different spectral ranges)

Each of these 118 dimensions has different magnitudes for a givenmaterial. Assuming that 134 different materials are used and combiningthem in groups of four to mip to a single material at a time generates alarge number of possible combinations. All dimensions of eachcombination need to be checked for a maximum or minimum value forcorrect standardization/normalization. Normalization enables neuralcomparisons using valid levels throughout system processing.

With M=134 materials and r=4 combined materials in one mipped material,there are

$\quad\begin{pmatrix}{r + M - 1} \\r\end{pmatrix}$combinations of M possible Material Coded Imagery (MCI) types, taken rways, with re-use irrespective of order. For example, if only materials51 and 76 are used, the following combinations would be made: {51, 51,51, 51}, {51, 51, 51, 76}, {51, 51, 76, 76}, {51, 76, 76, 76}, and {76,76, 76, 76} since order does not matter. There are thus

$\begin{pmatrix}{4 + 134 - 1} \\4\end{pmatrix} = {\begin{pmatrix}137 \\4\end{pmatrix} = {14,043,870}}$input patterns to be considered to obtain the normalization maximum andminimum values for the example embodiment.

FIG. 4 is a flowchart outlining the MCI normalization process. While thebelow-described steps are described as occurring in a particularsequence for convenience, it is noted that the order of variousoperations may be changed from embodiment to embodiment. It is furthernoted that various operations may occur simultaneously or may be made tooccur in an overlapping fashion. The process starts in S402 where a setof N MCIs are loaded. Next, in S404, for each dimension the MCIs areadded and averaged. Control continues to S406.

In S406, if an average for a particular dimension exceeds a storedmaximum value, then the stored maximum value is replaced with theaverage value. In S408, if an average for a particular dimension fallsbelow a stored minimum value, then the stored minimum value is replacedwith the average value. In S410, a determination is made as to whetherall MCI combinations have been considered. If all MCI combination havenot been considered, then control jumps back to S402 where the next setof N MCIs are loaded; otherwise, control continues to S420 where allpossible (in this example 14,043,870) combinations are created and theglobal normalization stored minimum and maximum values are adjusted suchthat: 0.0≤MCI combined value dimension≤1.0.

Returning again to FIG. 1, after the normalization circuitry 130 hasdetermined the normalization values for the MCI data combinations, thephase 1 training circuitry 140 performs a number of training operationsupon the neural model 126 using standardized/normalized MCI data 124.

Phase 1 training is a supervised training technique, in that the inputpattern also has a specific desired output result or classification thatis presented to the network. The network learns both the input patternand its classification. In this example, each output/classification is amaterial ID from a set of the pre-existing infrared materials.

However, in other embodiments each output/classification can vary to be,for instance, a material ID from a subset of the pre-existing infraredmaterials or from an expanded set of “synthesized materials.” Insynthesized-materials mode, new materials may be made by the neuralnetwork 126 to provide intermediate materials that better representcombinations of material in a mip, the mip being a combination ofmultiple materials that is being represented by a single material.During phase 1 training, neurons are generated/spread across the neuralnetwork 126 to represent the combined feature sets of the incoming setof r materials, through the operation of the neural network. The neuralnetwork 126 will automatically create neurons when needed, in order torepresent the combined materials, when they do not sufficiently resonatewith any of the existing material-coded neurons. These new neurons willrepresent a new material that may be used as the mipped material. Eachnewly-created material would be a hyper-point that exactly representsthe material properties.

In a non-limiting example, the neural network 126 is a Fuzzy AdaptiveResonance Technology network, or “Fuzzy ART.” The particular networkparadigm used in this system has additional features that: allow thestoring of an MCI number or set of numbers in an individual neuron,tagging it (without using a FuzzyARTMAP system); allow the learningcapabilities of specific neurons to be enabled or disabled; reading outthe internal neuron prototype values as a solution, as the final MCIparameter values after de-normalization; the hybridsupervised/unsupervised training methodology used to train the neuralnetwork with the phase 1 and phase 2 training is also novel; varying thestrength of the contribution of a certain dimension to the vigilancecalculation is also novel. Such a neural network paradigm normally wouldcreate neurons to represent the input patterns and then will findmatches to the learned patterns by calculating the amount of resonancedue to proximity of the input pattern to each of the stored prototypesthat represent the learned information. In this case where the system isre-using the same MCI IDs for all mipping, the Fuzzy ART network isseeded with only neurons that have their hyper-dimensional centers atlocations corresponding to settings of the infrared material parametersfor a single or set of MCI codes. Then, as each mipping combination ispresented to the network, one of the seeded MCI neurons will have thehighest resonance, and will be selected as being the closestrepresentation of the presented set of MCI numbers.

During phase 1 training, patterns are learned with high vigilance(vigilance=1.0) in order that the neural network 126 learn the inputsexactly. In this example, there are 118 physical parameters for each MCIcode, with some of them spectral in nature. As this phase of trainingproceeds, a training pattern and the name of the material it representsare presented to the neural network 126 so that it “learns” the patterncompletely in one neuron, and puts the respective ID of the material ona list it maintains for each neuron. If two input patterns have the sameexact weights then both of their IDs are added to a list for a singleneuron.

The resulting exemplars that are produced by phase 1 training consist ofhyper-points in multidimensional neural memory, as is depicted in FIG. 6(memory 610). Neural net training allows the neural memory 610 tocontain all the learned information for the input patterns {620, 630,640}. For the present embodiment, the neural memory is a 118-dimensionhypercube, where each dimension consists of a floating-point numericalrange of [0 . . . 1], which is assured by the previously discussednormalization of MCI data. Accordingly, each dimension fits into a [0 .. . 1] range without going above 1 or below 0, and thus no informationis lost.

In the phase 1 training, the input patterns {620, 630, 640} are appliedone at a time, with all dimensions in one pattern being applied at oncein parallel. Neurons are created for each new pattern that is presented,as long as it is not a duplicate of a previously-presented pattern.Since each phase 1 pattern is an exemplar, a single ‘crisp’ hyperpoint,and is meant to just create and code its own neuron, the resultingartifacts in the neural memory are represented as hyper-points. In oneembodiment, the hyper-points are 118-dimensional points. In FIG. 6, itis seen that training pattern 630 creates neuron 635 (tagged withidentifier 637 and containing the ID of the training pattern 630) whichin this case is labeled “2”. Similarly pattern 620 creates neuron 625,coding it with its pattern and tagging it 627 with its name “1.”However, because the material named “17” was identical to the materialnamed “1,” material “17” also is represented by neuron 625, and its IDis added to the list of material codes represented by that neuron. Allneurons are created in this way and there is no spare neuron.

In the manner of Fuzzy Adaptive Resonance Theory neural networktraining, the input pattern is complement encoded, as follows:

Input pattern is a, where: a=(a₁, a₂, . . . , a_(M))

In this mipping example, M=118 for 118 dimensions. The input pattern iscomplement encoded to get the neural net input pattern. The complementeda₁ isa ₁ ^(c)=1−a ₁;  Eq. (1)and the input vector becomes:

$\begin{matrix}\begin{matrix}{I = \left( {a,a^{c}} \right)} \\{= {\left( {a_{1},a_{2},\ldots\mspace{14mu},a_{M},a_{1}^{c},a_{2}^{c},\ldots\mspace{14mu},a_{M}^{c}} \right).}}\end{matrix} & {{Eq}.\mspace{14mu}(2)}\end{matrix}$

Since M=118 in the application, the total number of dimensions in I is2*M=236. Initially, there are no neurons. Since the vigilance is set sohigh, the neural network 126 will only code hyper-points (as oppose tohyper-rectangles) as may have otherwise occurred when using a lowervigilance. Vigilance, ρ, is a Fuzzy ART term that is defined in thefollowing manner:

$\begin{matrix}{\rho = \frac{{I\bigwedge w}}{I}} & {{Eq}.\mspace{11mu}(3)}\end{matrix}$where the Λ operator means to take the fuzzy Min of the two quantities,in this case it refers to taking the fuzzy Min of each dimensionseparately in the I

w term. The fuzzyMin of a scalar isx

y=min(x,y)  Eq. (4)

The norm brackets |·| indicate a sum across of the entire vector, as in|x|=Σ _(j=1) ^(M) x _(j)  Eq. (5)where the number of components in the vector is indicated by M.

The weight vector, w, describes the long-term memory template that isstored in each neuron. It consists of a pair of vectors, u and v,describing the ‘lowest’ corner and the ‘highest’ corner of thehyper-rectangle that has been learned by that neuron to represent a setof data in the training set, where

$\begin{matrix}\begin{matrix}{w = \left( {u,v^{c}} \right)} \\{= \left( {u_{1},u_{2},\ldots\;,u_{M},v_{1}^{c},v_{2}^{c},\ldots\;,v_{M}^{c}} \right)}\end{matrix} & {{Eq}.\mspace{11mu}(6)}\end{matrix}$

At phase 1 programming, the maximum vigilance, 1.0, occurs only for apattern that exactly matches a stored template. The only way to achieveρ=1.0 is if the neuron only has a single point and that the incominginformation is in exactly the same location of the point. For example,if the input pattern isa=(0.4,0.45)  Eq. (7)representing a single point in a 2-d coordinate system, as seen in FIG.7 as the single point 702, then the input pattern is:

$\begin{matrix}\begin{matrix}{I = \left( {a,a^{c}} \right)} \\{= \left( {0.4,0.45,0.6,0.55} \right)}\end{matrix} & {{Eq}.\mspace{11mu}(8)}\end{matrix}$

If this programs the neuron such that it is the only pattern representedby the neuron, then w=I, and

$\begin{matrix}{\frac{{I\bigwedge w}}{I} = {\frac{\left( {0.4,0.45,0.6,0.55} \right)}{\left( {0.4,0.45,0.6,0.55} \right)} = {\frac{2}{2} = 1}}} & {{Eq}.\mspace{11mu}(9)}\end{matrix}$

Any other point would not have a vigilance of 1.0. For instance if{hacek over (a)}=(0.4,0.46)  Eq. (10)yielding

$\begin{matrix}\begin{matrix}{\overset{\Cup}{I} = \left( {\overset{\Cup}{a},{\overset{\Cup}{a}}^{c}} \right)} \\{= \left( {0.4,0.46,0.6,0.54} \right)}\end{matrix} & {{Eq}.\mspace{11mu}(11)}\end{matrix}$and using the same w having the weights from having had been trainedwith the pattern I from Eq. (8), which is:

$\begin{matrix}\begin{matrix}{w = \left( {u,v^{c}} \right)} \\{= \left( {0.4,0.45,0.6,055} \right)}\end{matrix} & {{Eq}.\mspace{11mu}(12)}\end{matrix}$the resulting vigilance is then:

$\begin{matrix}{\rho = {\frac{{\overset{\Cup}{I}\bigwedge w}}{\overset{\Cup}{I}} = {\frac{\left( {0.4,0.45,0.6,0.54} \right)}{\left( {0.4,0.45,0.6,0.55} \right)} = {\frac{1.99}{2} = 0.995}}}} & {{Eq}.\mspace{11mu}(13)}\end{matrix}$and the neuron will not expand to code the new information because thevigilance has to equal or exceed the vigilance threshold of 1.0 in phase1, but in this case only equals 0.995.

FIG. 5 is a flowchart outlining phase 1 training. While thebelow-described steps are described as occurring in a particularsequence for convenience, it is noted that the order of variousoperations may be changed from embodiment to embodiment. It is furthernoted that various operations may occur simultaneously or may be made tooccur in an overlapping fashion. The process starts in S502 wherevigilance of a fuzzy ART is set to one. In S504, a training pattern(i.e., MCI data for a given material) is loaded/input and normalized. InS506, the training pattern along with its respective ID is applied tothe Fuzzy ART. In S510, a determination is made as to whether thetraining pattern is unique or redundant. If the training pattern isunique, control continues to S530; otherwise, control continues to S520.

In S530, a new neuron is created to represent the pattern loaded in S504and tagged with a pattern ID. Control continues to S540.

In S520, the ID of the non-unique/redundant pattern loaded in S504 isadded to a tag list of an appropriate existing neuron. Control continuesto S540.

In S540, a determination is made as to whether there are morepatterns/MCIs to be learned. If there are more patterns, control jumpsback to S504; otherwise, the operation of FIG. 5 stops.

Again returning to FIG. 1, after the phase 1 circuitry 140 has completedits initial training, the phase 2 circuitry 150 then performs a secondset of training operations.

Phase 2 Training:

After the neuron exemplars have been created as hyper-points, phase 2training is performed in order to expand the representation of theneurons with exposure to all the possible combinations of materialstaken r at a time. At this point, new materials may be created ifsynthesis of materials is desired. However, the use of existingmaterials will be described first.

In Phase 2 training, the vigilance threshold, ρ_(t), is reduced (e.g.,ρ_(t)=0.6) so that the existing neurons can update their weight vectorsto encompass a range of values in the hyper-dimensional classificationspace of the neural network 126, instead of just being coded as a singleexemplar hyper-point. In practice, the calculated vigilance of a patternwith respect to a given neuron's long term weights, w, has to exceed thevigilance threshold before that neuron may be modified.

Again, the large set of combined materials is generated, with N=thenumber of materials and r=the number of materials to combine with re-usebut irrespective of order. With N=134 materials and r=4 combinedmaterials in one material being mipped, there are

$\begin{pmatrix}{r + N - 1} \\r\end{pmatrix} = {14,043,870}$input patterns to be considered.

The input patterns must be normalized before they are applied to theneural net. The normalization values per dimension that were foundearlier can be used again in this step to ensure that all the inputs arenormalized in the same way, to allow correct comparisons to be made inthe neural memory.

A substantial difference in this phase of training versus that in phase1 is that no new neurons are allowed to be created when operating inexisting-materials mode. In contrast, when operating in a materialsynthesis mode, new neurons may be created.

In the phase 2 existing-materials only training mode, materialcombinations are presented one by one to the neural network 126, and theneural network 126 tests the activation of all the neurons to find theneuron with the highest activation. Activation, T, is defined as

$\begin{matrix}{T = \frac{{I ⩓ w}}{\beta + {w}}} & {{Eq}.\mspace{11mu}(14)}\end{matrix}$where β is a choice parameter that helps to add hysteresis in the choiceto avoid starting up new neurons that are too close to previouslylearned neurons, and a generally used value is β=0.01.

When the neuron having the highest activation is determined, the neuralnetwork 126 tests the vigilance of the neuron in representing the newpattern. If the vigilance surpasses the vigilance threshold, then theneuron is adjusted to include the new information by storing the fuzzymin of each long-term weight dimension versus the input-patterndimension value. If the vigilance value is not satisfied, then thenext-most-active neuron is tested, and on and on until there are no morepre-committed, already programmed, neurons left, and then either a newneuron is created, if operating in the appropriate mode, or the nextinput pattern is tried for programming the neural network 126 while thecurrent pattern is dropped. Having a dropped pattern may acceptablebecause, in the neural net use stage, the activations are checked again,and most distributed neurons will have activity to any input pattern,except for the single case with a neuron having a hyper-point at allzero responding to an incoming pattern of all ones.

Finding the activation follows as:

$\begin{matrix}\begin{matrix}{T = \frac{{I ⩓ w}}{\beta + {w}}} \\{= \frac{{\left( {0.3,0.33,0.7,0.67} \right) ⩓ \left( {0.4,0.45,0.6,0.55} \right)}}{\beta + 0.4 + 0.45 + 0.6 + 0.55}} \\{= \frac{\left( {0.3,0.33,0.6,0.55} \right)}{0.01 + 0.4 + 0.45 + 0.6 + 0.55}} \\{= \frac{0.3 + 0.33 + 0.6 + 0.55}{2.01}} \\{= 0.88557}\end{matrix} & {{Eq}.\mspace{11mu}(15)}\end{matrix}$

If this neuron was chosen as having the highest activation, then thevigilance will be calculated, as in:

$\begin{matrix}\begin{matrix}{\rho = \frac{{I ⩓ w}}{I}} \\{= \frac{{\left( {0.3,0.33,0.7,0.67} \right) ⩓ \left( {0.4,0.45,0.6,0.55} \right)}}{0.3 + 0.33 + 0.7 + 0.67}} \\{= \frac{\left( {0.3,0.33,0.6,0.55} \right)}{2}} \\{= 0.89}\end{matrix} & {{Eq}.\mspace{11mu}(16)}\end{matrix}$

If the threshold vigilance is 0.6, then this vigilance of 0.89 is higherthan the threshold vigilance, and thus the neuron can represent the newinput pattern. Accordingly, the new pattern is coded into the neuron. Itcan be coded slowly if the learning rate is low, but if the learningrate is set very high, it may learned completely the first timepresented. The new stored long-term weight vector for this neuron, withfast learning, is then

$\begin{matrix}\begin{matrix}{w = {I ⩓ w}} \\{= \left( {0.3,0.33,0.6,0.55} \right)} \\{= \left( {u,v^{c}} \right)}\end{matrix} & {{Eq}.\mspace{11mu}(17)}\end{matrix}$

The extents of the rectangle represented by w can be found from itsrepresentation in the form of (u, v^(c)), where:u=(0.3,0.33)v=(0.6,0.55)and hence

$\begin{matrix}{v = \left( {{1 - 0.6},{1 - 0.55}} \right)} \\{= \left( {0.4,0.45} \right)}\end{matrix}$having a graphical representation as shown in FIG. 10 where a rectangle1002 is shown encompassing the area within which a point would have thehighest activation, from u 1020 to v 1010.

This process continues through all the combinations of materials. If aneuron is a close-enough match, it is coded with the new information. Insome cases the new combination may fall directly inside thehyper-rectangle for an existing neuron, and no changes would have to bemade to the neuron. However, on other cases, there will be nosufficiently-closely-matching neuron, and no neuron will be programmedassuming that the existing-material only mode is used.

In contrast, if the operational mode is set such that new neurons (andthus new intermediate materials created) can be created, then a newneuron would be created. The amount of acceptable vigilance can beadjusted to ensure that only a certain number of total neurons arecreated in phase 2, so that the total number of IR materials, originaland synthetic, stays within a predetermined or otherwise desiredboundary.

FIG. 9 illustrates the result of presenting the unsupervised trainingpatterns to the neural network 126 in phase 2 training. This type oftraining can be considered to be that of a self-organizing (orunsupervised) training contrasting with the supervised trainingperformed in phase 1. In the example depicted in FIG. 9, no new neuronswere created, and the neural network 126 is forced to contain all of thenew learned information within the material-coded neurons that wereoriginally created in phase 1. A set of P patterns (item 900) ispresented to the neural network 126, where

${= \begin{pmatrix}{r + M - 1} \\r\end{pmatrix}},$and r=number of materials combined, with M=total number of materials,where the materials may be combined with re-use irrespective of order.In the case where M=134, and r=4 materials, then P=14,043,870 materialsto be presented to the neural net.

FIG. 9 depicts the combination of an instance of Material 7 (item 970),another instance of Material 7 (item 972), an instance of Material 21(item 974), and an instance of Material 3 (item 976). In this case r=4and there are four materials added (item 980) together and averaged bydividing by r (item 990). This creates material 950, which becomes (item995) one of set of P materials 900.

As the set of P materials 900 is presented to the neural network 126 oneby one, neurons may resonate with the input pattern. In the vernacularof Fuzzy ART, the resonance occurs when the vigilance of the matchbetween the neuron long-term memory weights and the incomingcompliment-encoded pattern is higher than the vigilance threshold.

The neuron with the highest activation (assuming a sufficiently highresonance) may expand to encompass the new neuron, using the fastadaptation method discussed above. Then what used to be a hyper-point635 may expand to a hyper-rectangle 920 if the input pattern 950resonates with that neuron 635 adding point 940 to expand the neuron'shyper-point into a hyper-rectangle 920.

The result of the phase 2 training is that the neurons are expanded tobetter represent the universe of material combinations that may bepresented to the network, thus helping to provide more accurateclassifications of material combinations. When used in the materialsynthesis mode, new neurons would be created to represent differentmaterial combinations. These new neurons would be created if there wasno neuron found that would satisfactorily represent, with high enoughvigilance, a given material combination when comparing it to the initialset of exemplar-trained neurons, or any of their combinations.

During the training with the combinations, to avoid any bias due to theorder in which the combinations are presented to the neural network, itmay be helpful to use a lower value for the learning rate, γ, instead ofthe value of γ=1 that lead to Eq. (18) below.w ^(new)=γ*(I

w ^(old))+(1−γ)w ^(old)  Eq. (18)

In this case, the phase 2 training would be repeated multiple times inorder to fully train the neurons with the completion of trainingdetected when either no neurons change or change below a smallthreshold.

Notice that for FIG. 9, the material codes 627, 637, 647, all stay thesame as the neuron learns new information. In this way only the originalmaterials will become the selected output neuron when the neural net isput into operation. In the case of when the neural network is used tosynthesize materials that have intermediate infrared properties, theneurons will be given new identification numbers that are not the sameas any of the identification numbers from the original set of materialspresented in phase 1 training.

FIG. 8 is a flowchart outlining phase 2 training. While thebelow-described steps are described as occurring in a particularsequence for convenience, it is noted that the order of variousoperations may be changed from embodiment to embodiment. It is furthernoted that various operations may occur simultaneously or may be made tooccur in an overlapping fashion. The process starts in S802 wherevigilance of a fuzzy ART is set to ρ, and the creation of new neurons isenabled or disabled. In S804, a set of R materials is combined to createa new pattern, and in S806, the new pattern is normalized.

In S808, the new pattern is applied to the neural network along with thepattern ID (optional) depending whether training is supervised orunsupervised, whereafter in S810 the neural network is trained to learnthe new pattern. In S820, a determination is made as to whether thereare more patterns to be learned. If there are more patterns, controljumps back to S804; otherwise, the operation of FIG. 8 stops.

Again returning to FIG. 1, after the phase 2 circuitry 150 has completedits initial training, the neural network 126 may be used by the mippingcircuitry 160.

How the Neural Network is Used:

Once the neural network 126 has been exposed to all the materialcombinations, then the weights will have been determined for all theneurons, and they may be saved to disk or some form of long-term,persistent memory. This weight file will only contain weight informationfor as many materials are allowed to exist in the network. For instance,if 134 materials were input to the network for training, even thoughmore than 14 million training patterns may have been produced, only amaximum of 134 neuron weight sets will be produced. These weights maythen be used in mipping by loading them into an instance of the sameneural network, and operating it in the run mode, also called the ‘use’,exploitation, or the discovery mode. In this process, a set of rmaterials are combined, where r=4 in one embodiment. The combination isapplied to the network. The neural network 126 checks all theactivations of all the neurons and chooses the one with the highestactivation as the best representation of the combined material, and thatMCI code is returned as the output, mipped, value for that materialcombination. In the case where there are multiple MCI codes, one thatmatches one of the MCI codes in the initial set of r combined MCI codesis returned as possible.

In the case where the material number returned must be the same as oneof the materials that is in the combination, only the neurons thatcontain one of the material IDs from the set of r materials presentedfor mipping are tested for activation.

In operation, a set of r materials is selected, combined and averagedand normalized for presentation to the neural network 126. The neuralnetwork 126 will then loop through all the neurons to find the neuronwith highest activation. If only original materials are considered, thenonly the neurons that were tagged with the material IDs from the rmaterials currently being mipped will be checked for activation, as thissaves many unneeded calculations.

If synthetic materials are used, then any material that is selected bythe neural net as being most similar to the combined incoming materialset will be examined. After the most active neuron is found, thematerial id will be returned to the calling program, but if the systemis only using pre-existing materials, then the material ID that isreturned will be one that matches one of the set of r materials thatwere combined before being applied to the network.

However, in other embodiments, each output/classification can vary tobe, for instance, a material ID from a subset of the pre-existinginfrared materials or from an expanded set of “synthesized materials.”In synthesized-materials mode, new materials may be made by the neuralnetwork to provide intermediate materials that better representcombinations of material in a mip, the mip being a combination ofmultiple materials that is being represented by a single material. A setof neurons would be automatically spread across the neural memory torepresent the combined feature sets of the incoming set of r materials,through the operation of the neural network. The neural network willautomatically create neurons when needed, in order to represent thecombined materials, when they do not resonate with any of the existingmaterial-coded neurons. Then these new neurons will represent a newmaterial that may be used as the mip material. Each newly-createdmaterial would be a crisp hyper-point that represents the materialproperties exactly. After being exposed to all the training combinationsin phase 2 unsupervised learning, the neuron will generally have ahyper-rectangular fuzzy prototype, or weight vector, w, consisting ofranges in different dimensions that need to be defuzzified in order todirectly be used as a crisp material having a single-valued, or crisp,value for each of the dimensions it uses to describe features orphysical aspects of the material. The defuzzification process for thisapplication could be to just take the average value in each of thedimensions. Each dimension will have a range that is specified by its uand v values, in their appropriate locations in the neuron weightvector. The low end of the range in that dimension is the value of u andthe upper end of the range is v=(1−v^(c)), with the output, synthesizeddefuzzified result, sdf, being

$\begin{matrix}{{sdf} = \frac{\left( {u + \left( {1 - v^{c}} \right)} \right)}{2}} & {{Eq}.\mspace{11mu}(19)}\end{matrix}$

The sdf values would be processed on a dimension by dimension basis, andused as the new material property in the resulting mip material. Analternative defuzzification method would be to find the mean of thedistribution of learned values in each dimension of the synthesizedmaterial, and use that as the defuzzified result for that dimension.

In some cases, one dimension of the neuron may have more meaning thananother, for instance if one dimension is temperature, while the otheris just a single spectral line from a 200-point spectrum. In these casesit can be helpful to decrease the vigilance weight of the minor propertyto be less than that of the major property, and hence influence thematching strength less for the lesser, or minor property, than for themajor property.

While the invention has been described in conjunction with the specificembodiments thereof that are proposed as examples, it is evident thatmany alternatives, modifications, and variations will be apparent tothose skilled in the art. Accordingly, embodiments of the invention asset forth herein are intended to be illustrative, not limiting. Thereare changes that may be made without departing from the scope of theinvention.

What is claimed is:
 1. A mipping system, comprising: processingcircuitry configured to: receive combinations of a plurality of pixels Nat a time, each pixel having material codes directed to respectivematerials of the pixels, where the material codes relate to infraredproperties of the respective materials sensed by a sensor of the mippingsystem, and N is a positive integer greater than 1; and train anartificial neural network having a classification space by providingrespective neurons for each unique combination of material codes, andcondition the artificial neural network so that the respective neuronsactivate when presented with their unique of material code combinations;calculate an average value of the material codes; replace a storedmaximum value with the average value when the average value exceeds thestored maximum value; replace a stored minimum value with the averagevalue when the average value falls below the stored minimum value; andnormalize the material codes using the replaced maximum and minimumvalues; train the artificial neural network starting with a vigilancesetting of a first value; and when the artificial neural network reachesa state in which no more new patterns are to be learned by theartificial neural network, adjust the vigilance setting of the firstvalue to a second value that is lower than the first value and greaterthan zero, and retrain the artificial neural network with the vigilancesetting of the second value.
 2. The system of claim 1, wherein eachmaterial code has P dimensions, where P is a positive integer greaterthan three, and the processing circuitry includes normalizationcircuitry that normalizes each material code such that all dimensions ofthe material codes have a range from 0.0 to 1.0.
 3. The system of claim1, wherein each material code has P dimensions, where P is a positiveinteger, and dimensions of the material codes include at least amaterial density, a material emissivity, a specific heat, a thermalconductivity, and a reflectivity for a plurality of different infraredspectral ranges.
 4. The system of claim 1, wherein the processingcircuitry is further configured to train the artificial neural networkstarting with the vigilance setting of the first value for an initialphase of the training during which a neural memory with multi-dimensionsstores the respective neurons for a plurality of input patterns that areapplied one input pattern at a time, the one input pattern being appliedat once in parallel with all the multi-dimensions of the neural memory.5. The system of claim 4, wherein the processing circuitry includes:phase one processing circuitry that trains the neural network using asupervised training technique and that employs a vigilance setting of1.0 so as to provide hyper-point classifications within theclassification space.
 6. The system of claim 5, wherein the processingcircuitry further includes: phase two processing circuitry that furthertrains the neural network after the phase one processing circuitryperforms its training, the phase two processing circuitry employinganother vigilance setting less than 1.0 so as to provide hyper-rectangleclassifications within the classification space.
 7. The system of claim6, wherein the phase two processing circuitry uses at leastnon-supervised training.
 8. A mipping method, comprising: receivingcombinations of a plurality of pixels N at a time, each pixel havingmaterial codes directed to respective materials of the pixels, where thematerial codes relate to infrared properties of the respective materialssensed by a sensor of a mipping system, and N is a positive integergreater than 1; training an artificial neural network having aclassification space by providing respective neurons for each uniquecombination of material codes, and conditioning the artificial neuralnetwork so that the respective neurons activate when presented withtheir unique of material code combinations; calculating an average valueof the material codes; replacing a stored maximum value with the averagevalue when the average value exceeds the stored maximum value; replacinga stored minimum value with the average value when the average valuefalls below the stored minimum value; and normalizing the material codesusing the replaced maximum and minimum values; training the artificialneural network starting with a vigilance setting of a first value; andwhen the artificial neural network reaches a state in which no more newpatterns are to be learned by the artificial neural network, adjustingthe vigilance setting of the first value to a second value that is lowerthan the first value and greater than zero, and retraining theartificial neural network with the vigilance setting of the secondvalue.
 9. The method of claim 8, wherein each material code has Pdimensions, where P is a positive integer greater than three, furthercomprising: normalizing each material code such that all dimensions ofthe material codes have a range from 0.0 to 1.0.
 10. The method of claim8, wherein each material code has P dimensions, where P is a positiveinteger, and dimensions of the material codes include at least amaterial density, a material emissivity, a specific heat, a thermalconductivity, and a reflectivity for a plurality of different infraredspectral ranges.
 11. The method of claim 8, wherein the neural networkuses a Fuzzy Adaptive Resonant Theory (“Fuzzy ART”) paradigm.
 12. Themethod of claim 11, wherein processing includes: phase one processingthat trains the neural network using a supervised training technique andthat employs a vigilance setting of 1.0 so as to provide hyper-pointclassifications within the classification space.
 13. The method of claim12, wherein processing further includes: phase two processing thatfurther trains the neural network after the phase one processingperforms its training, the phase two processing employing a-anothervigilance setting less than 1.0 so as to provide hyper-rectangleclassifications within the classification space.
 14. The method of claim13, wherein the phase two processing uses at least non-supervisedtraining.
 15. An imaging device, comprising: circuitry configured toprocess a plurality of input pixels taken N at a time using apre-trained artificial neural network to create an output pixel for eachN input pixels, wherein the input pixels and the output pixel allinclude infrared information produced by a sensor, each pixel having arespective material code chosen from material codes of a classificationspace, wherein N is a positive integer more than 1, wherein (i) anaverage value of the material codes is determined, (ii) a stored maximumvalue is replaced with the average value when the average value exceedsthe stored maximum value and a stored minimum value is replaced with theaverage value when the average value falls below the stored minimumvalue, (iii) the material codes are normalized using the replacedmaximum and minimum values, the pre-trained artificial neural network istrained initially with a vigilance setting of a first value, when thepre-trained artificial neural network reaches a state in which no morenew patterns are to be learned by the artificial neural network, thevigilance setting of the first value is adjusted to a second value thatis lower than the first value and greater than zero, and the pre-trainedartificial neural network is retrained with the vigilance setting of thesecond value.
 16. The device of claim 15, wherein the input pixels andthe output pixel all include more than three dimensions.
 17. The deviceof claim 15, further comprising circuitry that assembles a plurality ofoutput pixels to create a mipped infrared image.
 18. The device of claim17, wherein the artificial neural network includes a Fuzzy AdaptiveResonant Theory (“Fuzzy ART”) paradigm.
 19. The device of claim 18,wherein each material code has P dimensions, where P is a positiveinteger, and dimensions of the material codes include at least amaterial density, a material emissivity, a specific heat, a thermalconductivity, and a reflectivity for a plurality of different infraredspectral ranges.
 20. An imaging method, comprising: receiving aplurality of pixels that each include infrared information produced by asensor and more than three dimensions, each pixel having a respectivematerial code chosen from material codes of a classification space; andprocessing the pixels using a pre-trained artificial neural network tocreate an output pixel having a material code of the classificationspace and the infrared information, wherein (i) an average value of thematerial codes is determined, (ii) a stored maximum value is replacedwith the average value when the average value exceeds the stored maximumvalue and a stored minimum value is replaced with the average value whenthe average value falls below the stored minimum value, and (iii) thematerial codes are normalized using the replaced maximum and minimumvalues, the pre-trained artificial neural network is trained initiallywith a vigilance setting of a first value, when the pre-trainedartificial neural network reaches a state in which no more new patternsare to be learned by the artificial neural network, the vigilancesetting of the first value is adjusted to a second value that is lowerthan the first value and greater than zero, and the pre-trainedartificial neural network is retrained with the vigilance setting of thesecond value.
 21. The method of claim 20, wherein the output pixelincludes more than three dimensions.
 22. The method of claim 21, furthercomprising assembling a plurality of output pixels to create a mippedinfrared image.
 23. The method of claim 22, wherein the artificialneural network includes a Fuzzy Adaptive Resonant Theory (“Fuzzy ART”)paradigm.
 24. The method of claim 23, wherein each material code has Pdimensions, where P is a positive integer, and dimensions of thematerial codes include at least a material density, a materialemissivity, a specific heat, a thermal conductivity, and a reflectivityfor a plurality of different infrared spectral ranges.